Importance sampling in reinforcement learning with an estimated behavior policy

Abstract In reinforcement learning, importance sampling is a widely used method for evaluating an expectation under the distribution of data one policy when has in fact been generated by different policy. Importance requires computing likelihood ratio between action probabilities target and those data-producing behavior this article, we study where are replaced their maximum estimate these observed data. We show general technique reduces variance due to error Monte Carlo style estimators. introduce two novel estimators that use expected values arise RL literature. find reduce methods, leading faster learning gradient algorithms more accurate off-policy evaluation. also provide theoretical analysis showing our new consistent have asymptotically lower than